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A Microscope Imaging System (CMIS) with intelligent 
controls is disclosed that provides techniques for scanning, 
identifying, detecting and tracking micrbscopic changes in 
selected characteristics or features of various surfaces 
including, but not limited to, cells, spheres, and manufac- 
tured products subject to difficult-to-see imperfections. The 
practice of the present invention provides applications that 
include colloidal hard spheres experiments, biological cell 
detection for patch clamping, cell movement and tracking, 
as well as defect identification in products, such as semi- 
conductor devices, where surface damage can be significant, 
but difficult to detect. The CMIS system is a machine vision 
system, which combines intelligent image processing with 
remote control capabilities and provides the ability to auto- 
focus on a microscope sample, automatically scan an image, 
and perform machine vision analysis on multiple samples 
simultaneously. 

2 Claims, 25 Drawing Sheets 








COMPUTER 



SOURCE 





U.S. Patent May 9,2006 


Sheet 2 of 25 


US 7,042,639 B1 




FIG-2 










FIG 







FIG-5 FIG- 







U.S. Patent May 9,2006 


Sheet 5 of 25 


US 7,042,639 B1 




TO FIG-7B 


FIG-7A 










U.S. Patent May 9,2006 


Sheet 6 of 25 


US 7,042,639 B1 



FROM FIG-7A 



FIG-7B 








U.S. Patent 


US 7,042,639 B1 



FIG- 




Sheet 8 of 25 


US 7,042,639 B1 


U.S. Patent May 9,2006 




U.S. Patent May 9,2006 


Sheet 9 of 25 


US 7,042,639 B1 





iws? 


■s^M 4j tl?S'«r s i&yj i u s 5#»s4ste 


'gmm 

$gj§& 


ggppl 


m4mm 

t* asg-MfA*!*! 




•:■'■ SI 


: .M ; •? 




U.S. Patent 


May 9, 2006 


Sheet 10 of 25 


US 7,042,639 B1 


136 

V 



TO FIG-14B 


FIG-14A 









U.S. Patent 


May 9, 2006 


Sheet 11 of 25 


US 7,042,639 B1 


136 


FROM FIG-14A 



FIG-14B 








U.S. Patent May 9,2006 


Sheet 14 of 25 


US 7,042,639 B1 


176 



TO FIG-19B 


FIG-19A 









U.S. Patent May 9,2006 


Sheet 15 of 25 


US 7,042,639 B1 


176 


FROM FIG-19B 



FIG-19B 










U.S. Patent May 9,2006 


Sheet 16 of 25 


US 7,042,639 B1 






US 7,042,639 B1 




U.S. Patent 


May 9, 2006 Sheet 19 of 25 


US 7,042,639 B1 



U.S. Patent 


May 9, 2006 


Sheet 20 of 25 


US 7,042,639 B1 


230 



FIG-27 







US 7,042,639 B1 






U.S. Patent 


May 9, 2006 


Sheet 23 of 25 


US 7,042,639 B1 


248 



FIG-32 







U.S. Patent May 9,2006 


Sheet 24 of 25 


US 7,042,639 B1 




U.S. Patent 


May 9, 2006 


Sheet 25 of 25 


US 7,042,639 B1 



FIG-38 



US 7,042,639 B1 


1 2 

IDENTIFICATION OF CELLS WITH A SUMMARY OF THE INVENTION 

COMPACT MICROSCOPE IMAGING 

SYSTEM WITH INTELLIGENT CONTROLS The present invention is directed to a Microscope Imaging 

System (CMIS) that utilizes intelligent controls and provides 


This application is a divisional application of co-pending 5 
parent case Ser. No. 10/645,999, which was filed on Aug. 21 , 
2003. 

ORIGIN OF THE INVENTION 10 

The invention described herein was made by an employee 
of the United States Government and may be used by or for 
the Government for governmental purposes without pay- 
ment of any royalties thereon or therefor. 15 

FIELD OF THE INVENTION 

The invention relates to a Compact Microscopic Imaging 2Q 
System (CMIS) with intelligent controls. Specifically, the 
invention relates to a CMIS that provides techniques for 
scanning, identifying, detecting and tracking microscopic 
changes and selected characteristics or features of various 
surfaces included, but not limited to, cells, spheres, and 25 
manufactured products subject to difficult-to-see imperfec- 
tions. The CMIS provides intelligence to allow experiments 
to be conducted without the need of constant or even human 
monitoring. 

30 

BACKGROUND OF THE INVENTION 

Microscope experiments are typically conducted by 
humans utilizing microscopes that provide enlarged images 35 
of minute objects or specimens being analyzed. The suc- 
cessful results of these experiments are dependent upon the 
concentration, endurance, and constant monitoring of a 
human operating the microscope. Sometimes the human 
operator of the microscope on a daily basis loses concen- 40 
tration, becomes fatigued as the day goes on, or becomes 
distracted with all these factors contributing to unsuccessful 
experiments. It is desired that a system be provided that 
allows experiments to be conducted using microscope, but 
without the need of intervention by a human. 45 

OBJECTS OF THE INVENTION 

It is a primary object of the present invention to provide 
a system that allows experiments to be conducted utilizing 50 
microscopes, but without the need of human intervention. 

It is another object of the present invention to allow 
experiments to be conducted without the need of human 
intervention and without the need of constant monitoring. 

It is a further object of the present invention to provide a 55 
system for analyzing microscope experiments automatically 
and remotely. 

It is a further object of the present invention to provide a 
system that utilizes intelligence and controls the operation of g0 
the system by the use of adaptive neural networks. 

Still further, it is an object of the present invention to 
provide for a system that scans to find areas of interest, 
focuses and acquires images automatically. 

In addition, it is an object of the present invention to 65 
provide a system to evaluate surfaces of devices, such as 
semiconductors, to ascertain the quality thereof. 


techniques for scanning, identifying, detecting and tracking 
microscopic changes in selected characteristics or features 
of various objects including, but not limited to, cells, spheres 
and manufactured products subject to difficult-to-see imper- 
fections. 

In one embodiment, an imaging system for examining 
specimens is provided comprising: a video microscope 
comprising a holder for specimens and a camera located so 
as to be focussed on the holder. The imaging system further 
comprises a light source operatively connected to the video 
microscope, and a robotic system operatively connected to 
and for positioning the video microscope. The imaging 
system further comprises a computer operatively connected 
to the camera and to the robotic system. The computer has 
operating programs comprising routines for providing 
machine vision techniques for scanning, identifying, detect- 
ing and tracking selected characteristics and features of the 
specimen being analyzed. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The features and advantage of the invention, as well as the 
invention itself, will be better understood with reference to 
the following description when considered in conjunction 
with the accompanying drawings, wherein like reference 
numbers designate identical corresponding parts thereof and 
wherein: 

FIG. 1 is a block diagram of the imaging system of the 
present invention; 

FIG. 2 is a flow chart of one embodiment of the present 
invention; 

FIG. 3 illustrates a colloid image related to the embodi- 
ment of FIG. 2 and shows horizontal growth; 

FIG. 4 illustrates a colloid image related to the embodi- 
ment of FIG. 2 and shows vertical growth; 

FIG. 5 illustrates a typical colloid image; 

FIG. 6 is a typical frame averaged image; 

FIG. 7 is composed of FIGS. 7A and 7B. which cumu- 
latively illustrate a flow chart for another method of the 
present invention; 

FIG. 8 illustrates the result of brightness slicing on a 
frame-average image related to the embodiment of FIG. 7; 

FIG. 9 illustrates an image of a dilation treated image 
combined with optimal threshold related to the embodiment 
of FIG. 7; 

FIG. 10 is an image treated with a find particle algorithm 
related to the embodiment of FIG. 7; 

FIG. 11 is an image illustrating the interface analysis 
associated with the embodiment of FIG. 7; 

FIG. 12 illustrates a typical colloid image showing related 
to vertical growth associated with the embodiment of FIG. 
7; 

FIG. 13 illustrates the image showing the interface analy- 
sis related to vertical growth associated with the embodi- 
ment of FIG. 7; 

FIG. 14 is composed of FIGS. 14A and 14B that cumu- 
latively illustrate a flow chart related to a further method of 
the present invention; 

FIG. 15 illustrates an out-of-focus image related to the 
method of FIG. 14; 

FIG. 16 illustrates Fast Fourier Transform (FFT) of an 
out-of-focus image related to the method of FIG. 14; 
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FIG. 17 illustrates an in-focus image related to the method 
of HG. 14; 

FIG. 18 illustrates a FFT of an in-focus image related to 
the method of FIG. 14; 

FIG. 19 is composed of FIGS. 19A and 19B that cumu- 5 
latively illustrate a flow chart related to a still further method 
of the present invention; 

FIG. 20 illustrates an image of a candidate specimen 
related to the method of FIG. 19; 

FIG. 21 illustrates a histogram that as received Sobel to 
filtering operation all related to the method of FIG. 19; 

FIG. 22 illustrates an image after receiving variance 
filtering related to the method of FIG. 19; 

FIG. 23 illustrates an image after receiving Sobel and 
variance filtering techniques all related to the method of 15 
FIG. 19; 

FIG. 24 illustrates an image after having all objects 
therein identified and related to the method of FIG. 19; 

FIG. 25 illustrates an image of a second candidate speci- 
men related to the method of FIG. 19; 20 

FIG. 26 illustrates the result of the image of FIG. 25 after 
having received the operation of the method of FIG. 19; 

FIG. 27 illustrates a flow chart related to another method 
of the present invention; 

FIG. 28 illustrates the original cell image related to the 25 
method of FIG. 27; 

FIG. 29 illustrates an image processed by the method of 
FIG. 27; 

FIG. 30 illustrates a image that had been labeled process 
by the method of FIG. 27; 30 

FIG. 31 illustrates an original image with the cells labeled 
and associated with the method of FIG. 27; 

FIG. 32 is a flow chart related to an additional method of 
the present invention; 

FIG. 33 is an image of an ideal case surface related to the 35 
method of FIG. 32; 

FIG. 34 illustrates a defective surface detected by the 
method of FIG. 32; 

FIG. 35 illustrates another defective surface detected by 
the method of FIG. 32; 40 

FIG. 36 illustrates an ideal surface of another semicon- 
ductor device related to the method of FIG. 32; 

FIG. 37 illustrates a second defective surface detected by 
the method of FIG. 32; and 

FIG. 38 illustrates another defective surface detected by 45 
the method of FIG. 32. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

50 

Referring to FIG. 1, there is shown a block diagram of 
Compact Microscope Imaging System 10 of the present 
invention particularly suited to perform microscope analy- 
sis. 

As will be further described, the Compact Microscope 55 
Imaging System (CMIS) 10, sometimes referred to herein 
simply as an imaging system, is a diagnostic tool with 
intelligent controls for use in space, industrial, medical, and 
security applications. The CMIS 10 can be used in-situ with 
a minimum amount of user intervention and it can be used 60 
to scan, find areas of interest, focus, and acquire images all 
automatically. 

The CMIS 10 provides machine vision techniques in 
order to achieve the goal of being an intelligent microscope 
capable of several automated tasks and able to reproduce 65 
human microscope system interactions with accuracy and 
repeatability. The CMIS 10 may be used to replace what 
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humans do with microscopes on a daily basis. Unlike human 
operation, the CMIS 10 will not lose concentration, become 
fatigued as the day goes on, or become distracted like a 
human might. 

A wide variety of applications that can take advantage of 
the CMIS 10 having intelligent controls for usage in fields 
such as materials science, fluids science, life sciences, 
biotechnology and production monitoring for quality control 
where human visualization of a cell or surface is difficult. 
These include experiments involve dendrite growth, trans- 
parent system coarsening and sintering, soil mechanics 
(including lunar and Martian experiments), colloid science, 
magneto and electro-rheological fluids, details of flow 
around bubbles, particle pushing and engulfment, foam 
behavior, protein and zeolite crystal growth, examination of 
living systems and microscopic or extremely small defects 
in high value parts. Large numbers of multiple cell experi- 
ments require microscopy for in-situ observations; this is 
feasible with the CMIS 10. 

In general, the CMIS 10 provides machine vision tech- 
niques characterized as follows: (1) adaptive thresholdings 
which optimally determines the best method to achieve the 
image contrast needed for image processing; (2) auto-focus 
which provides the ability to focus on a sample automati- 
cally utilizing a Cartesian robot system; (3) auto-imaging 
scanning which enables a microscope to scan left, right, up, 
down, back, and forth within a sample in order to find the 
object of interest; (4) object identification and classification 
which enables the finding, classifying and labeling of 
objects in a predetermined area of interest; (5) motion 
detection which enables the observing and quantifying of the 
movement of objects in a pre-determined area of interest; 
and (6) transition mapping which enables the detection of 
small transitions between groups of cells used for micro- 
scope experiments such as order/disorder, large objects/ 
small objects, light/dark regions and movement/non-move- 
ment. Transition mapping is particularly helpful in 
determining the exact transition location of the quasi-zone, 
which is in-between two phases in a microscope image. 

To further appreciate the'benefits of the present invention 
the CMIS 10 will be further described hereinafter for four 
(4) methods which are: (1) interface detection in colloidal 
hard spheres systems; (2) auto-focus for compact micro- 
scope systems; (3) cell identification and classification for 
biomedical applications and (4) surface defect classification. 
The overall operation may be further described with refer- 
ence to FIG. 1. 

The CMIS 10 comprises a video microscope 12, which, in 
turn, comprises a holder 14 and a camera 16 located, so as 
to be focused on the holder 14. The CMIS 10 further 
comprises light source 18 that is operatively connected to 
the video camera 12 via signal path 20 and provides for 
illumination of any specimens being held in the holder 14. 
The CMIS 10 further comprises a robotic system 22 
response to electrical signals and which is operatively con- 
nected to the video microscope 12 by way of control path 24 
and preferably by way of platform 26 upon which the video 
microscope 12 resides. Hie robotic system 22 preferably 
further comprises a micro-positioner 22A. 

The CMIS 10 further comprises a computer 28 that is 
operatively connected to the camera 16 by way of an image 
acquisition board 30 and signal path 32. The computer 28 
further provides electrical signals and is operatively con- 
nected to the robotic system 22 by way of a robotic control 
board 34 and signal path 36. 

The operation of the computer 28 is controlled by its 
operating programs 36, which are comprised of machine 
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vision routines 38, algorithms 40, and neural network rou- 
tines 42. The operating programs 36 provide visual verifi- 
cation of the results of the performed testing, to be 
described, on display terminal 44, via signal path 46. 

The video microscope 12 is preferably compact and may 5 
be of the type that are made commercially available by 
various companies such as Fuji, Sony, Edmund Scientific 
and Zeiss. The camera 16 is preferably comprised of coupled 
devices and the camera 16 feeds the data to the acquisition 
board 30 by way of the signal path 32. The camera 16 to 
provides an output representative of the electronic image of 
the specimen being held in holder 14. 

The light source 18 is preferably a fiber-optic, which may 
be of the type that are made commercially available by 
various companies, such as Cuda Products, Thor Labs and 15 
Semiconductor International. The robotic system 22 is pref- 
erably of a Cartesian type and provides three ( 3 ) dimen- 
sional positioning of the video microscope 12 which, in turn, 
provides three ( 3 ) dimensional positioning of the specimen 
being held in the holder 14, as well as the camera 16. The 20 
robotic system 22 may be of the type that are made com- 
mercially available by various companies, such as Parker 
Hanifin, Dadel Inc., and Physic Instruments. The operating 
programs 36 residing in the computer 28 further comprise 
routines (known in the art) for controlling the robotic system 25 
which, in turn, control the positioning of the video micro- 
scope 12. 

The algorithms 40, to be further described hereinafter, 
operatively cooperate with routines that are part of a 
machine vision routines that provide for scanning, identify- 30 
ing, detecting and tracking selected characteristics and fea- 
tures of the many specimens capable of being analyzed by 
the CMIS 10. 

The neural network routines are of a conventional type 
that consists of a set of elements that start out connected in 35 
a random pattern, and based upon operational feedback, are 
molded into patterns required to generate the required results 
to be described hereinafter. These neural network routines 
operatively control the robotics system 22, as well as diag- 
nose and provide image processing and pattern recognition 40 
related to the machine vision routines 38. 

Adaptive neural networks are essentially probabilistic 
computations or techniques that are used to adapt neural 
networks for automated control tasks. A basic neural net- 
work approach is to accept several inputs (N) and based on 45 
an error relationship produce a finite set of outputs (M). The 
difficulty in using basis neural networks is that the results 
can be unpredictable and the neural network may fail to 
yield an optimal solution if conditions suddenly change. 
However, when adaptive neural networks are implemented 50 
as for the present invention, the “best case” scenario is 
optimized by a set of learning behaviors built into the neural 
network. Just as humans adapt to a changing environment, 
so can an adaptive neural network. Such is the case in the 
practice of the present invention with regard to the technique 55 
of auto-focus, auto-image scanning and the detection of 
defects. Rather than have several inputs mapped to a finite 
set of outputs (M), an adaptive neural network takes several 
inputs (N) and maps them to a set of outputs (M+l), with the 
(+1) referring to a new case that can be optimized/defined in 60 
the future. 

The practice of the present invention uses inputs or 
features that a human would use to identify, detect and 
classify objects, cells and areas of interest. These are rep- 
resented by the initial sets of images that are received. The 65 
practice of the invention uses image processing techniques 
and creates a relational database of the features identified by 
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the humans. These features serve as a baseline input feature 
set and are used to train/teach the adaptive neural network 
preferably used in the practice of the invention. The practice 
of the invention develops a machine vision technique that is 
used to identify the input feature set, but also has the ability 
to learn how to adapt to various changing inputs to produce 
an optimal output. For example, let’s take a group of cells 
that have 4 defects in each cell. A standard neural network 
technique, as well as an adaptive neural network technique, 
will identify only cells with 4 defects. Now for example, 
take a group of cells with 7 defects in them. A standard 
neural network technique will output no cells identified or 
label all cells as having only 4 defects, which would be a 
gross misrepresentation of the data. However, an adaptive 
neural network technique practiced by the present invention 
outputs cells with 4, 5, 6, or 7 defects since it has the ability 
to reason or adapt to the changing number of defects. One 
of the machine vision routines is related to interface detec- 
tion of colloidal hard spheres systems. 

Microscopic hard spheres in a colloid suspension have 
been used for studying atomic phase changes from disorder 
to order or from liquid to solid. It is my understanding that 
current microscope image analysis techniques are not able to 
distinguish between these densely packed phases within 
microscope images, which have similar grayscale value 
properties and are mainly characterized by degrees of ran- 
domness or order. The current technique for identifying the 
phased change boundaries involves manually identifying the 
transitions, which is very time consuming and inefficient. 
The present invention provides a method that overcomes 
these difficulties. More particularly, the present invention 
provides a technique that accurately detects the interface of 
colloidal hard spheres experiments automatically and with- 
out human intervention and may be further described with 
reference to FIG. 2. 

FIG. 2 illustrates an overall flow chart for a routine 48 for 
interface detection of colloidal hard spheres. The routine 48 
is comprised of steps represented by segments 50, 52, 54, 56, 
58, and 60 shown in FIG. 2, some of which serve as program 
segments. , . 

In general, a colloid system, examined by the practice of 
the present invention, consists of two phases, a solid or 
ordered phase and a liquid or disordered phase. There is a 
quasi-zone in between these two phases, which has to be 
analyzed in order to determine an accurate interface. 
Depending on the orientation of the sample, the practice of 
the present invention provides an interface technique to 
analyze the following two cases: (1) horizontal crystal 
growth from left to right as shown in FIG. 3, and (2) vertical 
crystal growth from bottom to top as shown in FIG. 4. 

If the solid/liquid interface is oriented horizontally (the 
crystal is growing across the screen), the rightmost portion 
of this object contains the contour of the solid/liquid inter- 
face generally identified by reference number 62 of FIG. 3. 
If the solid/liquid interface is oriented vertically (the crystal 
is growing up the screen), the top portion of this object 
contains the contour of the solid/liquid interface generally 
identified by reference number 64 of FIG. 3. 

Based on kinetic theory, it is that matter is made of 
molecules in continual random motion. In a crystalline or 
solid state, these molecules are held in place by attractive 
forces and tend to vibrate around a fixed position. In a liquid 
state, the molecules have a greater range of motion. With this 
concept in mind, it is expected that the molecules in the 
liquid portion of the system exhibit a higher degree of 
motion than those in the solid. This higher degree expecta- 
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tion allows the present invention to use frame averaging to 
visually separate solid from liquid regions. 

The method of FIG. 2 averages a series of frames per- 
formed by one of algorithms 40 by summing each individual 
pixel over each frame in the series and dividing by the 
number of frames. In a colloidal suspension, the spheres in 
the “liquid” region have more freedom of movement than 
the spheres in the solid region. When multiple frames are 
averaged, the liquid region tends to become blurry or grayed 
out while the solid region maintains a higher level of 
contrast and may be represented by the below expression: 


Oil - ^ /»’ 

n 

i,j: row and column position of the pixel in a frame 
(image) 

n: number of frames 

I: input pixel, 

O: output pixel 

FIGS. 5 and 6 give examples of a typical image and the 
averaged image over a series of frames, wherein the liquid 
and solid portions of FIG. 5 are respectively generally 
identified by reference numbers 66 and 68 and wherein the 
liquid and solid portions of FIG. 6 are respectively generally 
identified by reference numbers 70 and 72. Note the loss of 
contrast in the liquid portion of the averaged image of FIG. 
6. The overall operation of the method of interface detection 
of colloid hard surfaces may be further described with again 
reference to FIG. 2. 

As seen in FIG. 2, the method for interface detection of 
colloidal hard sphere is initiated by segment 50, which 
signifies the placing of the colloidal hard sphere specimen, 
having a solid/liquid interface to be examined, in the holder 
14. Upon such placement, the method of FIG. 2 passes 
control to program segment 52, by way of signal path 74. 

The program segment 52 causes the video microscope 12 
to generate an electronic image, serving as a frame of the 
colloidal specimen. Upon such generation by program seg- 
ment 52, control is passed to program segment 54, by way 
of signal path 76. 

The program segment 54 examines the frame of the 
electronic image to determine if the solid/liquid interface has 
a horizontal or vertical growth previously discussed. Upon 
completion, program segment 54 passes control to program 
segment 56, by way of signal path 78. 

The program segment 56 generates a series of frames of 
electronic images of the examined specimen and then passes 
control to program segment 58, by way of signal path 80. 

The program segment 58 performs averaging of the series 
of frames to provide an imaged averaged over the series of 
frames so as to separate the solid/liquid portions of the 
electronic image. Upon completion, program segment 58 
passes control to program segment 60, via signal path 82. 

The program segment 60 examines, by displaying on 
terminal 44, the averaged image to identify the liquid 
portions by blurred or grayed out regions of the averaged 
image and to identify the solid portions as clear regions with 
contrast therein. 

Another embodiment for providing a routine for interface 
detection of colloidal hard surfaces, may be further 
described with reference to FIG. 7, which is composed of 
FIGS. 7A and 7B, wherein FIG. 7A illustrates the overall 
method 84 comprised of elements 86, 88, 90, 92, 94, 96, and 
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98, and wherein FIG. 7B illustrates the overall method 84 as 
further comprising elements 100, 102, 104, 106, and 108. 
The program 84 of FIG. 7 is different than the program 48 
of FIG. 2 in that it provides for a brightness slicing tech- 
5 nique. 

The brightness slicing technique is used to create a very 
high-contrast image that specifically highlights a particular 
object of interest and works well in separating objects that 
have either light or dark values from a background with 
to intermediate values. In the averaged image shown in FIG. 6, 
previously discussed, the ordered or solid portion of the 
image is made up of light and dark values, while the 
disordered or liquid portion of the image has intermediate 
gray values. As will be further described, the brightness 
is slicing technique is performed on a series of frame-averaged 
images and when applied, the result will be a solid region 
highlighted in white against a black background, thereby 
making it quite easily to determine the solid-liquid interface. 
FIG. 8, to be further described illustrates a typical colloid 
20 image utilizing the brightness slicing technique. It should be 
noticed how there is now a distinct separation pattern from 
the solid and liquid regions of the image respectively 
generally identified by reference numbers 110 and 112. 

Brightness slicing employed by the method of FIG. 7 is a 
25 double binary contrast enhancement operation, serving as 
another algorithm 40, where pixel brightness’s below a 
lower threshold and above an upper threshold are set to 
white, while brightness’s between the two thresholds are set 
to black. This results in the liquid or blurry region being set 
30 to black while the solid region is set to white and may be 
represented by the below expression: 


Oij =0, a, < lij < a 2 
35 

= 1, otherwise 

a x : lower threshold value. 

40 a 2 : upper threshold value. 

The values of the quantities a t and a 2 are dependent upon 
the experiment being conducted and more importantly, a 
light or a dark background. The range of the quantities, a! 
and a 2 , can be between O (black) and 255 (white). The 
45 brightness slicing technique may be further described with 
again reference to FIG. 7. 

As seen in FIG. 7A, the overall routine 84 for interface 
detection of colloidal hard surfaces employing the bright- 
ness slicing technique is initiated by segment 86 which 
50 signifies the placing of colloidal hard sphere specimen in 
holder 14. Upon such completion, the method 84 passes 
control to program segment 88, by way of signal path 114. 

Program segment 88 causes the video microscope 12 to 
generate an electronic image, serving as a frame, of the 
55 colloid specimen and then passes control to program seg- 
ment 90, via signal path 116. 

Program segment 90 examines the electronic image to 
determine the original size of the pixel representing articles 
of the colloid specimen. Original size determination is 
60 utilized by program segment 98 to be further described 
hereinafter. Upon completion, program segment 90 passes 
control to program segment 92, by way of signal path 118. 

Program segment 92 examines the electronic image and 
determines solid/liquid interface as manifesting a horizontal 
65 or vertical crystal growth. Upon completion, program seg- 
ment 92 passes control to program segment 94, by way of 
signal path 120. 
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Program segment 94 generates a series of frames of the 
electronic image. Upon completion, program segment 94 
passes control to program segment 96, by way of signal path 

122 . 

Program segment 96 performs averaging of the frames to 
provide an image average over the series of frames so as to 
visually separate the solid and liquid portions of the elec- 
tronic image. Upon completion, program segment 96 passes 
control to program segment 98, by way of signal path 124. 

Program segment 96 takes into account the fact that due 
to the solid region containing some of the same intensity 
values as the liquid regions, some erosion of the particle may 
occur. Program segment 98 compensates for the erosion by 
dilating the particles to approximate the original size. 

After the brightness slice operation performed by program 
segments 88, 90, 92 and 94, the solid portion of the region 
is made up of many individual white pixels. A Find Particles 
algorithm, part of the algorithms 40, to be further described 
hereinafter, will only find particles that are 8-connected. A 
perfectly 8-connected object has a continuous perimeter that 
can be traced by traveling in 8 directions (up, down, left, 
right, and diagonals) and at least one pixel separating the 
object from the image edge. For the method of FIG. 7, it is 
desired to dilate the white pixels to ensure that they will 
form a connected mass. In one practice of the present 
invention, the best dilation technique was a 7x7 low pass 
filter with 0 at the center and larger weights away from the 
center as shown in Table 1 . 


TABLE 1 


Vsi 

V 32 

Vi6 

Vs 

Vis 


Vsi 

V$2 

V 16 

Vs 

>/4 

Vs 

Vi 6 

Vi2 

VlS 

v» 

V* 

Vi 

Vi 

Vs 

Vis 

!/s 

Vi 

Vi 

0 

Vi 

Vi 

Vs 

VlS 

Vs 

>/4 

'A 

Vi 

Vs 

Vis 

Vi2 

Vl6 

Vs 

Vi 

Vs 

Vis 

V32 

'/Si 

Vi2 

VlS 

Vs 

Vis 

Vi2 

Vsi 


The dilation produces a thin strip along the border of the 
image that is blacked out to ensure the solid region is 
separated from the image edge. 

With reference to Table 1, it is seen the dilation produces 
a square matrix of weights of size m (in this case 7). This 
weighting is applied to the neighborhood of the current pixel 
(the current pixel is centered) and is given as follows: 


“'ll Wll ■■■ Wim 
W21 W 2 2 ... 


W >ml 


Upon completion, program segment 98 passes control to 
program segment 100, shown in FIG. 7B, via signal path 

126. 

Program segment 100 performs a threshold algorithm 
previously discussed with reference to brightness slicing. 
Upon completion, program segment 100 passes control to 
program segment 102, by way of signal path 128. 

Program segment 102 receives the dilated threshold 
treated electronic image of program segment 100. Program 
segment 102 then examines and displays the dilated thresh- 
old treated electronic image to visually identify solid and 
liquid portions, each manifesting a separate pattern and 
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preferably also employs a fine particle technique that scans 
the image and identifies every non-background object. 

The find particle technique, serving as one of the algo- 
rithms 40, searches a grayscale image for non-background 
5 particles using an eight directional search (N, NE, E, SE, S, 
SW, W, NW) known in the art. The centroid of each particle 
is found using an intensity weighted center of mass algo- 
rithm also known in the art and part of the algorithms 40. 

The program segment 102 provides the examined dilated 
10 and threshold treated electronic image, and displays such on 
the display terminal 44 as represented by the image shown 
in FIG. 10. The image of FIG. 10 may be further enhanced 
by performance of program segments 104 and 106. Upon 
completion, program segment 102 passes control to program 
is segment 104, by way of signal path 129. 

Program segment 104 superimposes a contrast line 130 on 
the interface shape as shown in FIG. 11. To determine the 
interface 130, the largest particle found is determined to be 
the solid region. Depending on the previously specified 
20 orientation (horizontal or vertical growth) the topmost pixel 
in each column or the rightmost pixel in each row is taken 
as the solid/liquid interface. Upon completion, program 
segment 104 passes control to program segment 106, by way 
of signal path 132. . 

25 Program segment 106 examines the contrast line to define 
data thereof. Upon completion, program segment 106 passes 
control, via signal path 134 to program segment 108. 

Program segment 108 stores the data in computer 28, in 
particular the database for the examined colloidal hard shape 
30 sphere residing in the holder 14. 

FIGS. 8, 9, 10, and 11 represent the images associated 
with the horizontal growth for the interface detection tech- 
nique employing brightness slicing. This interface detection 
technique employing brightness slicing is also applicable to 
35 the vertical growth case and is shown in FIGS. 12 and 13. 

It should now be appreciated that the present invention 
provides two related techniques for the detection of the 
interface of the colloid hard sphere experiments. Because of 
the practice of the present invention, experiments that used 
40 to take weeks or even months to analyze manually can now 
be accomplished in a manner of minutes or hours for a 
longer experiment. 

Another method of the present invention related to auto- 
matically focusing on a specimen may be further described 
45 with reference to FIG. 14 composed of FIGS. 14A and 14B, 
each showing portions of the overall method 136. More 
particularly, FIG. 14A shows the overall method 136 for 
automatically focusing on a specimen as comprised of 
elements 138, 140, 142, 144, 146, and 148. Similarly, FIG. 
50 14B shows the overall method 136 for automatically focus- 
ing on a specimen as further comprised of elements 150, 
152, 154, and 156. 

As seen in FIG. 14A, the routine 136 for automatically 
focusing on a specimen starts off with attaching a micro- 
55 positioner 22A, shown in FIG. 1, to a Cartesian robotic 
system 22. The micro-positioner 22A preferably has three 
stages: coarse; medium; and fine. The micro-positioner 22A 
has a vertical level having a top limit which causes the 
camera of the video microscope 12 to be situated by pre- 
60 determined distance, which is the furthest distance possible 
from the target that is, the specimen that is held in holder 14. 
This predetermined distance depends on the type of robotic 
system being used. It can be as small as a millimeter and as 
long as a few feet. The practice of the invention used a 
65 system with a maximum distance of 1 foot and a system with 
a maximum distance of 2 inches and both worked well. The 
micro-positioner 22A, preferably has ten separate steps for 
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each stage and the step size for the coarse stage is about 10 
mm. Upon completion of segment 138, the method of FIG. 
14 passes control to segment 138, via signal path 158. 

Segment 140 signifies that the specimen to be examined 
is placed in a holder and then segment 140 passes control to 
program segment 142, by way of signal path 160. 

Program segment 142 causes the video microscope 12 to 
generate an electronic image comprises spatial frequencies, 
wherein any spatial frequency represents the rate of bright- 
ness transition. 

Images generated by the video microscope 12 are made 
up of spatial frequencies. A spatial frequency is the rate of 
brightness transition (the rate at which spatial details cycle 
from light to dark and back to light). The Fast Fourier 
Transform is one method used to decompose an image into 
its fundamental frequency components. There is a magni- 
tude and a phase value associated with each frequency 
component. For each image, a focus value is computed by 
taking the Fast Fourier Transform (FFT) of the largest 
possible portion of the center of the image. After the FFT 
operation, the image moves from the spatial domain to the 
frequency (or Fourier) domain. When an image is displayed 
in the frequency domain, the magnitudes of the frequency 
components that make up the image are also displayed. 

For each pixel in the original image, there is one value 
displayed in the frequency transform image. The display of 
the magnitudes in the frequency domain is arranged sym- 
metrically around the center of the image, with horizontal 
frequencies on the horizontal axis, vertical frequencies on 
the vertical axis, and diagonal frequencies off the axes. The 
zero- frequency point of the display is in the center. Frequen- 
cies increase outward from the center of the image to the 
maximum rate at the outermost point. The magnitude of 
each frequency component is indicated by the darkness of 
the pixel at each location in the image: the darker the pixel, 
the greater the magnitude of the corresponding spatial 
frequency. 

Upon completion, program segment 142 passes control to 
program segment 144, by way of signal path 162. 

Program segment 144 performs a Fast Fourier Transform 
on the electronic image so as to decompose the electronic 
image into its fundamental components with each compo- 
nent having magnitude and phase qualities. The FFT per- 
formance transforms the electronic image from a spatial 
domain to a frequency domain. Upon completion, program 
segment 144 passes control to program segment 146, by way 
of signal path 164. 

Program segment 146 examines the FFT electronic 
images by displaying the magnitude of the frequency com- 
ponents. A blurry, poorly focused image (see FIG. 15) is 
made up of low frequencies positioned near the center of the 
FFT. This results in a compact FFT pattern, with dark pixels 
near the center and a lot of white area (see FIG. 16). A 
correctly focused image of the same scene (see FIG. 17) will 
contain higher frequencies, resulting in a dispersed FFT 
pattern, with dark pixels spread over a wider area and less 
white background (see FIG. 18). Upon completion, program 
segment 146 passes control to program segment 148, by way 
of signal path 166. 

Program segment 148 positions the vertical position of a 
micro-positioner 22A to its top level so that the camera is 
located away from the specimen by its first predetermined 
distance which is also the maximum distance in which the 
camera can be positioned away from the specimen. Upon 
completion, program segment 148 passes control, via signal 
path 168, to program segment 150 shown on FIG. 14B. 
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Program segment 150 positions by separate steps, pref- 
erably 10, the micro-positioner 22A so that the camera 
moves downward and toward the specimen and a predeter- 
mined number of step sizes, preferably 10 mm, associated 
5 with the coarse stage of the micro-positioner, while at the 
same time examining that the FFT pattern being displayed 
becomes further dispersed, in a manner as shown in FIG. 18, 
as the camera moves downward by each of the coarse steps. 
Upon completion, program segment 150 passes control to 
to program segment 152, by way of signal path 170. 

Program segment 152 determines the sum of a magnitude 
of the FFT pattern while at the same time determining that 
the sum of the magnitude of the FFT pattern decreases as the 
camera moves downward at each coarse step. The magni- 
15 tude over the FFT are summed for each image as the image 
becomes more focussed. The position where the focus value 
is at lowest is considered to be most in-focus. It is preferred 
that the step size of 10 mm be decreased by a factor of 10 
and that program segments 150 and 152 be repeated. During 
20 such adjustments, the micro-positioner 22A moves such that 
the current-in-focus positioner is centered in a new range. 
Upon completion, program segment 152 passes control to 
program segment 154, by way of signal path 172. 

Program segment 154 repeats program segments 150 and 
25 152, but sets the micro-positioner 22A to its medium stage 
before such repeating. Upon completion, program segment 
154 passes control to program segment 156, by way of 
signal path 174. 

Program segment 156 repeats program segments 150 and 
30 152, but now sets the micro-positioner 22A to its fine stage 
before such repeating is performed. 

Upon completion of program segment 156, the method 
136 produces an image that is ideally focussed to its maxi- 
3 _ mum contrast. 

The positioning of the micro-positioner 22A may be 
accomplished by a neural network routine appropriately 
programmed so that it sets its associated elements in a 
random manner and, based upon operational feedback these 
40 elements are molded into the pattern required to provide the 
required results, that is, an ideally focused image with 
maximum contrast. 

Another embodiment of the present invention that com- 
prises a routine for cell identification and classification 
45 includes the matrix/statistics of one or more cells, may be 
further described with reference to FIG. 19 composed of 
FIGS. 19A and 19B, wherein FIG. 19A shows the overall 
method 176 as comprised of the elements 178, 180, 182, 
184, 186, 188 and, similarly, FIG. 19B shows the overall 
50 method 176 as further comprised of elements 190, 194, 196, 
198, 200, 202, and 204. 

Cell identification for biomedical applications is done all 
around the world on a daily basis by thousands of humans 
looking through microscopes for hours at a time. The related 
55 technique of the present invention utilizes human informa- 
tion and incorporates that into an intelligent machine vision 
technique in order to develop a system capable of using only 
the best techniques for cell identification. The present inven- 
tion provides a technique for the automatic detection of cell 
60 candidates for patch clamping, known in the art. This 
technique automatically identifies all possible cells in an 
image (Good & Bad). This technique also provides metrics/ 
statistics, known in the art, on cells such as intensity- 
weighted center of mass (centroid), major axis length, minor 
65 axis length, area, roundness, smoothness, elongation, shape, 
partial cells, and cell orientation. FIG. 20 illustrates an 
example of a potential patch clamping candidate. The 
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method 176 for cell identification may be further described 
again with reference to FIG. 19A. 

As seen in FIG. 19A, the overall method 176 starts with 
segment 178 wherein the one or more cells serving as a 
specimen are placed in the holder. Upon completion, seg- 
ment 178 sequences to program segment 180, by way of 
signal path 206. 

Program segment 180 generates a series of electronic 
images of the specimen and then passes control to program 
segment 182, by way of signal path 208. 

Program segment 182 generates a histogram having bars, 
which are proportional to the values of the intensity of the 
series of electronic images. In order to identify and classify 
cells, it is necessary to pre-process the images to allow 
isolation of the cells from their background. For the patch 
clamping images related to program segment 176, two 
parallel pre-processing paths are provided. The first path 
related to program segment 182 is a histogram illustration 
shown in FIG. 20. Upon completion, program segment 182 
passes control to program segment 184, by way of signal 
path 210. 

Program segment 184 performs a Sobel filtering operation 
of the histogram so as to highlight the edges of the changes 
in the density of the histogram. The Sobel filter highlights 
the edges by finding changes in intensity thereof. The Sobel 
filtering technique is known in the art. FIG. 21 shows the 
image after receiving the histogram’s treatment and the 
Sobel filtering operation. Upon completion, program seg- 
ment 184 passes control to program segment 186, by way of 
signal path 212. 

Program segment 186 performs a threshold algorithm 
previously described with reference to the brightness slicing 
technique of the present invention. 

The threshold algorithm, serving as one of the algorithms 
40, eliminates small intensity changes in the image provided 
to program segment 186. Program segment 186 clearly 
identifies particle boundaries, but does not always preserve 
cell connectivity (cell parameters often have gaps). This is 
corrected by allowing program segment 186 to pass control, 
via signal path 214, to program segment 188. 

Program segment 188 causes the video microscope 12 to 
generate an electronic image, which is now used for the 
second path of the overall process 176 of FIG. 19. Upon 
completion, program segment 188 passes control, via signal 
path 216, to program segment 190 shown on FIG. 19B. 

Program segment 190 performs a variance filtering opera- 
tion, serving as one of the algorithms 40, to produce an 
image thereof shown in FIG. 22. As the filter passes over the 
image, the variance, s, is computed as 


s2 = ^Z (x ‘-^ 2 ’ 
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where N is the number of pixels and x is the average in each 
neighborhood. Upon completion, program segment 190 
passes, via signal path 218, control to program segment 194. 

Program segment 194 examines the variance of the neigh- 
5 borhood of the pixels. If the neighborhood of the center pixel 
has a low variance, the area is basically constant and the 
center of the pixel is set to black. More particularly, and with 
reference to FIG. 23, if the center pixel having a lip-like 
10 shape has a low variance, then the center of the lip-like shape 
is set to black. If the neighborhood has a high variance, 
intensity values are changing over the neighborhood and the 
center pixel is left unchanged. This approach preserves the 
connectivity of the cells and has no corruption of cell date, 
is but is also includes some of the background as part of the 
cell. Upon completion, program segment 194 passes control 
to program segment 196, by way of signal path 220. 

Program segment 196 combines the treated histogram 
program segment 186 with the colored or uncolored center 
20 pixel of program segment 194 that results in an image shown 
in FIG. 24. Upon completion, 196 passes control to program 
segment 198, by way of signal path 222. 

Program segment 198 performs a find pixel technique to 
25 identify non-background objects of the cbmbined histogram 
of program segment 196 in a manner as previously described 
with reference to FIG. 7. 

Program segment 198, as well as other program segments 
of the present invention, may employ a clean technique that 
30 cleans the image using a center-deleted spike filter modified 
to remove noise not associated with particles. The clean 
technique is arranged to work on images with a black 
background. The mask size designates the extent of the 
neighborhood around the current pixel that will determine its 
35 output value. A clutter threshold, known in the art, represents 
the cutoff for the sum of pixel values lying within the mask 
region (except the center pixel). If the sum is less than the 
clutter threshold, the center pixel will be set to 0. A signifi- 
cance threshold represents the cutoff for a given pixel value 
40 to be considered significant. If the center pixel is below the 
significance threshold, it 'will be set to 0. When cleaning a 
binary image, the significance threshold is not necessary 
since all pixels are either white (255) or black (0), so it is set 
to 0. 

45 Upon completion, program segment 198 passes control to 
program segment 200, by way of signal path 224. 

Program segment 200 performs an overlay technique, 
known in the art, on the combined histogram to block out 
background objects and provides an image such as that 
50 shown in FIG. 24. Upon completion, program segment 200 
passes control to program segment 202, by way of signal 
path 226. 

Program segment 202 analyzes the non-background 
objects to produce matrix of the specimen being held in the 
holder. 


TABLE 2 


#id 

X 

y 

major 

minor 

area 

elong 

round 

smooth theta 

thin 

prtl 

1 

517.1 

18.3 

14.9 

8.8 

117 

0.589 

2.169 

0.184 

98 F 

F 

2 

289.3 

274.9 

124.5 

66.5 

4882 

0.534 

0.178 

0.143 

22.6 F 

F 

3 

84.2 

310 

3.5 

2.7 

16 

0.776 

0.88 

0.154 

31.7 F 

F 

4 

388.4 

313.4 

6.4 

3.2 

24 

0.495 

0.93 

0.183 

37 F 

F 

5 

227.2 

330.7 

16.7 

11.2 

155 

0.67 

0.967 

0.16 

114.9 F 

F 

6 

602.9 

369.1 

4.7 

3.2 

21 

0.674 

1.817 

0.115 

135 F 

F 
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TABLE 2-continued 


#id 

X 

y 

major 

minor 

area 

elong 

round 

smooth theta 

thin 

pit] 

7 

594.5 

373.9 

11.9 

6.1 

62 

0.515 

0.697 

0.19 

137.6 F 

F 

8 

91.1 

400.9 

37.8 

9.4 

202 

0.249 

0.232 

0.161 

42.2 F 

F 

9 

151.9 

451.3 

16.4 

7.6 

99 

0.462 

0.535 

0.169 

26.1 F 

F 

10 

158.7 

463.4 

13.7 

9.9 

106 

0.723 

0.898 

0.174 

43.5 F 

F 


Based on the information listed in Table 2, program 
segment 202 can classify an ideal patch clamping candidate 
by its major axis, minor axis, area, elongation, roundness, 
smoothness, theta (orientation with respect to a coordinate 
system), thinness, and whether the object is completely (F) 
contained in the field of view or only partially contained 
(partial). Program segment 202 passes control, via signal 
path 228, to program segment 202. 

Program segment 204 compares the produced metrics of 
Table 2 against known metrics to identify and classify the 
one or more cells being held in the holder 14. 

The result from our technique practicing the method 176 
of FIG. 2 and starting with the image of FIG. 25 yielded the 
following Table 3. 


further described with reference to FIG. 27 for an overall 
program 230 having elements 232, 234, 236, and 238. 

As seen in FIG. 27, the overall method starts with segment 
232 which signifies one or more cells, serving as a specimen, 
are placed in the holder 14. Upon completion, segment 232 
gives control to program segment 234, by way of signal path 
240. 

Program segment 234 generates an electronic image, such 
as that shown in FIG. 28, of the one or more cells, each 
having a dark outer ring. Upon completion, program seg- 
ment 234 passes control to program segment 236, by way of 
signal path 242. 

Program segment 236 examines the dark outer ring of 
each cell to determine the shape and size of fhe one or more 


to 


15 


20 


TABLE 3 


#id 

X 

y 

major 

minor 

area 

elong 

round 

smooth theta 

thin 

prtl 

1 

72.1 

89.4 

191.8 

126.7 

12491 

0.661 

0.077 

0.143 

137 F 

T 

2 

597.3 

33.1 

49.9 

44.9 

844 

0.899 

0.18 

0.162 

1.2 F 

T 

3 

564.9 

11.8 

6.8 

3.7 

27 

0.537 

0.908 

0.203 

40.3 F 

F 

4 

575.4 

28.2 

17.3 

9.3 

114 

0.536 

0.449 

0.1 

83.3 F 

F 

5 

559.1 

21.3 

4 

2.5 

15 

0.615 

0.943 

0.159 

80.8 F 

F 

6 

629.1 

41 

3.4 

2.3 

13 

0.676 

0.99 

0.205 

103.3 F 

T 

7 

628.6 

51.2 

12 

4 

38 

0.333 

0.489 

0.144 

90 T 

T 

8 

594.8 

97.5 

16 

14 

175 

0.876 

0.884 

0.168 

4.7 F 

F 

9 

168.7 

107.6 

4 

4 

19 

1 

1.775 

0.167 

OF 

F 

10 

93.1 

238.7 

247.1 

117.3 

14243 

0.475 

0.093 

0.139 

44.7 T 

F 

11 

598.9 

128.9 

35.8 

14.6 

277 

0.409 

0.323 

0.185 

99.4 F 

F 

12 

614.3 

132.3 

22.1 

5.5 

121 

0.25 

0.361 

0.161 

76.5 F 

F 

13 

623.2 

146.4 

5.4 

1.9 

15 

0.346 

0.619 

0.135 

74 F 

F 

14 

629.6 

158.2 

3.2 

1 

7 

0.302 

0.752 

0.188 

74.5 F 

T 

15 

342.4 

223.3 

121.6 

88.4 

6417 

0.727 

0.301 

0.141 

160.6 F 

F 

16 

623.8 

185.2 

6.7 

4.1 

26 

0.618 

0.965 

0.2 

58.3 F 

F 

17 

600.6 

371 

5.1 

3.8 

21 

0.744 

1.805 

0.188 

125.8 F 

F 

18 

593.4 

376.9 

6.7 

5 

31 

0.739 

1.269 

0.203 

137.4 F 

F 

19 

100.8 

392.3 

8 

4.6 

34 

0.572 

0.994 

0.147 

36 F 

F 

20 

149.5 

453 

9.4 

5 

50 

0.535 

0.845 

0.125 

26.6 F 

F 

21 

155.4 

464.9 

5.4 

2.2 

15 

0.396 

0.75 

0.25 

5.7 F 

F 


By looking at the results listed in Table 3, it is clear that 
cell 15 of Table 3 is the ideal candidate because it most 50 
closely matches the metrics identified for an ideal patch 
clamping candidate cell. This is a significant advancement in 
the field of patch clamping because we now have the ability 
to automatically scan through a sample specimen and iden- 
tify ideal cell candidates with no human interaction. The 55 
technique has proven to be more efficient over time than an 
actual human who has to look at each and every cell sample 
over a period of hours. 

A second example resulting from the patch clamping 
technique of the present invention is illustrated in FIG. 26. 

Another method of the practice of the present invention 
provides for a routine that identifies one or more cells in an 
area of interest, which cell is closest to the center of the area 
interest, the nearest neighbors of each cell within the area of 65 
interest and tracks movements of the cells within the area of 
interest routine for cell identification. The method may be 


cells. By concentrating on the dark outer ring of each cell, 
it allows for determination of the shape and size of the cell, 
but prior to the examination the cells may have an image 
such as that of FIG. 29. Upon completion, program segment 
236 passes control to program segment 238, by way of 
signal path 244. 

Program segment 238 performs an intensity weighted 
center of mass technique to determine the position of each 
cell within the area of interest. The intensity weighted center 
of mass technique is known in the art and serves as one of 
the algorithms 40. 

The intensity weighted center of mass technique deter- 
mines the position of each cell to each other. The program 
segment 238 then labels the cell based on the information of 
each center with a labeled process image such as that shown 
in FIG. 30 and the original image with the cells labeled 
shown in FIG. 31. The center of each cell is identified in 
FIG. 30 by reference number 246A. The partial particles 
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shown in FIG. 31 by reference number 246B can be clas- 
sified as such and then perimeters within the image can be 
determined. 

It should now be appreciated that the practice of the 
present invention provides a procedure that seeks to main- 5 
tain high accuracy and repeatable techniques for cell iden- 
tification in the medical industry. It is contemplated that this 
technique also reduces, by a significant amount, the amount 
of lost time human error typically creates. 

Another embodiment provided by the machine vision to 
techniques of the present invention comprises a routine for 
evaluating surfaces of devices, such as semiconductor 
devices and is shown in FIG. 32, and an overall program 248 
comprised of elements 250, 252, 254, and 256. 

The technique illustrated by FIG. 32 utilizes an ideal or 15 
representative surface and treats this ideal surface as the 
standard for evaluating remaining samples. Each surface to 
be analyzed is compared to the standard and if it meets 
certain tolerance metrics, it will be approved for use. The 
goal in this technique is to categorize good or acceptable 20 
surfaces vs. bad or unacceptable surfaces. By classifying the 
surface defects, the present invention labels and categorizes 
each defect as it appears in an area of interest. The surface 
classification technique works by isolating any defects on a 
given surface and transforming the pixel information into an 25 
objective measure of surface quality that can be compared to 
the ideal case. 

As seen in FIG. 32, the program 248 is initiated by a 
program segment 250, which provides an electronic image 
of a desired surface of a device. The electronic image of 30 
program segment 252 possesses an intensity representative 
of the critical value of the desired surface. Upon completion, 
program segment 250 passes control to program segment 
252, by way of signal path 258. 

Program segment 252 represents the placement of the 35 
device in the holder so that its surface faces the camera of 
the video microscope 12. Upon completion, program seg- 
ment 252 passes control to program segment 254, by way of 
signal path 260. 

Program segment 254 generates an electronic image of 40 
the surface of the device in the holder. This electronic image 
includes a morphological gradient operation (known in the 
art), which is performed on the image to find feature edges. 
This operation creates an edge-enhanced image of the 
boundaries of the features in the image, such as that shown 45 
on the lower portion of FIG. 34. The outlines represent the 
portions of the image where the steep dark-to-bright and 
dark-to-dark transition occur. Then a center deleted spike 
filter, previously discussed with reference to FIG. 19, is 
applied to remove one-pixel duration variations and small 50 
features of the pixel. The pixels in the resulting image are 
summed to provide a measure of surface quality. Surface 
quality for each case is then compared to the critical value 
represented by the ideal surface. FIG. 33 represents a 
smooth or ideal surface, whereas FIGS. 34 and 35 show 55 
examples of the same type of surface exhibiting defects. The 
surface quality value for each case is compared to the critical 
value represented by the ideal case of FIG. 33. If a surface 
has an intensity value greater than the ideal surface as in the 
cases of FIGS. 34 and 35, it may be classified as being 60 
defective. The critical value can be adjusted for various 
types of surfaces. 

A second ideal surface is shown in FIG. 36, which 
represents an ideal case, and FIGS. 36 and 37 show defect 
samples of the same type of surface related to FIG. 36. To 65 
accommodate all of this analysis, program segment 254 
passes control to program segment 256, by way of signal 
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path 262. Program segment 256 completes all the compari- 
sons of the intensity of a desired surface and the surface 
being evaluated. 

The surface classification technique represented by FIG. 
27 allows an ideal surface to be stored in an image database 
and retrieved for comparison of similar surfaces that can be 
used for in-line process inspection or for an automatic 
design system to accept or reject a surface based on the ideal 
surface. 

It should now be appreciated that the practice of the 
present invention provides a CMIS that can conduct experi- 
ments using microscopes without the need of constant 
monitoring. Further, the CMIS allows for the analysis of 
microscopic experiments automatically and remotely. Still 
further, the CMIS provides a utilization of intelligence 
allowed by adapted neural networks to cause the desired 
feed-back resulting in accurate and repeatable results. 

It should be still further appreciated that the CMIS scans, 
finds areas of interest, focuses and acquires images auto- 
matically. It should be further appreciated that the CMIS 
allows for the evaluation of surfaces of devices, such as 
semiconductors to be accurately and quickly determined. 

The invention has been described with reference to pre- 
ferred embodiments and alternates thereof.lt is believed that 
many modifications and alteration to tie embodiments as 
described herein will readily suggest themselves to those 
skilled in the art upon reading and understanding the 
detailed description of the invention. It is intended to include 
all modifications and alterations insofar as they come within 
the scope of the present invention. 

What I claim is: 

1. A method for examining specimens comprising the 
steps of: 

a) providing a video microscope having a holder for 
specimens and a camera located so as to be focused on 
said holder and providing an output representative of 
the electronic image of said specimen; 

b) providing a light source operatively connected to said 
video microscope for illuminating said specimens 
being held in said holder; 

c) providing a robotic system responsive to electrical 
signals and operatively connected to and for position- 
ing said video microscope; 

d) providing a computer having a display terminal and 
operatively connected to said output of said camera and 
providing said electrical signals to said robotic system, 
said computer having operating programs comprising 
routines for providing machine vision techniques for 
scanning, identifying, detecting and tracking and dis- 
playing on said display terminal selected characteristics 
and features of said specimen, wherein said routines for 
providing machine vision techniques comprise a rou- 
tine for cell identification and classification including 
metrics/statistics of one or more cells within an elec- 
tronic image comprising the steps of: 

a) placing said one or more cells comprising a specimen 
in said holder; 

b) causing said video microscope having a camera to 
generate a series of electronic images of said specimen; 

c) generating a histogram having bars which are propor- 
tional to the values of the intensity of the series of 
generated electronic images; 

d) performing a Sobel filtering operation of said histo- 
gram so as to highlight the edges of the changes in 
intensity of said histogram; 

e) performing a threshold algorithm having a predeter- 
mined cut-off value on said histogram having said 
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Sobel filtering so as to eliminate relatively small inten- 
sity changes of said histogram having said Sobel fil- 
tering; 

f) causing said video microscope to generate an electronic 
image of said specimen; 

g) performing a variance(s) filtering operation of said 
electronic image of said step f) wherein the variance, s, 
is computed as: 


where N is the number of pixels occupying neighborhoods 
and x is the average number of pixels in each neighborhood 
and wherein said pixels include a center pixel; 

h) examining the variance of the neighborhood of the 
center pixel and establishing the color of its center and 
if the variance is below a first predetermined variance 
then setting the color of its center pixel to a color black, 
but if the variance is above a second predetermined 
variance the color of its center pixel is left unchanged; 
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i) combining said histogram of said step e) with the 
colored or unchanged colored center pixel of step h); 

j) performing a find particles technique to identify all the 
non-background objects in said combined histogram 

5 and colored or uncolored center pixel; 

k) performing an overlap on said combined histogram and 
colored or uncolored center pixel of said step j) so as to 
block out background objects thereof; 

l) analyze said non-background objects to produce metrics 

10 thereof; and 

m) comparing said produced metrics against known met- 
rics so as to identify and classify said one or more cells. 

2. The method according to claim 1, wherein said metrics 

of said identified and classified one or more cells comprise 

15 their major axis, minor axis, area, elongation, roundness, 

theta which represents an orientation with respect to a 

predetermined coordinate, thinness, and whether the object 

of the respective one or more cells being examined is within 

the field of view of said camera of said video microscope or 
20 r 
only partially contained within the field of view of said 

camera of said video microscope. 

***** , 



